Multimedia Resource Discovery

Resource discovery is more than just search: it is browsing, searching, selecting, assessing and evaluating, ie, ultimately accessing information. Giving users access to collections is one of the defining tasks of a library. For thousands of years the traditional methods of resource discovery have been facilitated by librarians: they create reference cards with meta-data that are put into catalogues (nowadays, databases); they also place the objects in physical locations that follow certain classification schemes and they answer questions at the reference desk. The advent of digital documents has radically changed the organisation principles; now it is possible to automatically index and search document collections as big as the worldwide web à la Google and browse collections utilising author-inserted links. It is almost as if automated processing has turned the traditional library access paradigm upside down. Instead of searching meta-data catalogues in order to retrieve the document, web search engines search the full content of documents and retrieve their meta-data, ie, the location where documents can be found. Not all manual intervention has been abandoned, though. For example, the Yahoo! directory is an edited classification scheme of submitted web sites that are put into a browsable directory structure akin to library classification schemes. Undoubtedly, however, it is the automated approaches that have made all the difference to the way the vast world-wide web can be used. While the automated indexing of text documents has been successfully applied to collections as large as the world-wide-web for over a decade now, multimedia indexing by content involves different, still less mature and less scalable technologies. Figure 1 shows a matrix of different search engines depending on the query (columns) and document repository (rows). Entry a) in this matrix corresponds to a traditional text search engine with a completely different technology than entry b), a system that allows you to express musical queries by humming a tune and that then plays the corresponding song; the three c) entries in Figure 1 correspond to a multi-modal video search engine allowing search by motion with example images and text queries, eg, find me video shots of the Houses of Parliament in London zooming into its tower’s clock face using the following example still image; in contrast to this d) could be a search engine with a query text box that returns BBC Radio 4 discussions. Section 2 summarises different basic technologies involved in multimedia search. Multimedia collections pose their very own challenges; for example, images and videos don’t often come with dedicated reference cards or meta-data, and when they do, as in museum collections, their creation will have been expensive and time-consuming. Section 3 explores the difficulties and limitations of automatically indexing, labelling and annotating image and video content. It briefly discusses the inherent challenges of the semantic gap, polysemy, fusion and responsiveness.